Skip to content

Filter modules before loading them and load the kept ones in one call - #2283

Merged
josevalim merged 2 commits into
elixir-lang:mainfrom
KirillKayumov:retriever-filter-early-batch-load
Nov 2, 2026
Merged

josevalim merged 2 commits into
elixir-lang:mainfrom
KirillKayumov:retriever-filter-early-batch-load

Conversation

@KirillKayumov

Copy link
Copy Markdown
Contributor

What

Two changes in ExDoc.Retriever, one in ExDoc.Autolink:

  1. filter_modules runs on the docs chunk, before the module is loaded and processed. Previously every module went through Code.ensure_loaded, abstract code translation, specs, callbacks and markdown parsing, and only then the predicate decided whether to keep the node. A filtered-out module now costs a Code.fetch_docs read and its Refs entries, nothing else. The predicate receives the chunk's metadata, which is what the :filter_modules docs describe ("the module and its metadata"); it no longer sees the :kind key the retriever adds to the node later.
  2. Kept modules are loaded with one Code.ensure_all_loaded/1 call. Code.ensure_loaded/1 per module from six Task.async_stream workers serializes on the code server; on a large project that was most of the retrieval time.
  3. Autolink keeps the filtered modules in a MapSet and checks MapSet.member? instead of Enum.any? over a list of ModuleNode structs for every module reference. The retriever returns the filtered modules as sorted names; ExDoc.Formatter.autolink/5 builds the set once.

Output is unchanged: the same pages, sidebar and search data, byte for byte apart from Makeup's per-run group prefix.

Why

On a private Elixir project with ~50 000 modules (~39 600 documented), measured locally with 6 schedulers:

run before after
filter_modules allow-list of 100 modules, modules already loaded by an earlier step 55–62 s 22 s
same allow-list, modules not preloaded (mix docs alone) 468 s 20 s
full build, 39 600 modules, not preloaded 473 s 102 s

The 468 s and 473 s cases are the per-module Code.ensure_loaded round trips; the project only avoided them by accident because an earlier task in the same VM had loaded every module. The allow-list case was additionally slowed down by the Enum.any? scan: with ~6 500 filtered modules a filtered build took 3.7× longer than an unfiltered one.

Notes for review

  • Code.ensure_all_loaded/1 exists since Elixir 1.15, the minimum mix.exs requires.
  • docs_from_dir/2 and docs_from_modules/2 now return {[ModuleNode.t()], [module()]}. Both are @doc false-level internals of a @moduledoc false module; the existing tests that matched {nodes, []} still pass unchanged. The filtered_modules autolink option is now a MapSet of names (tests updated).
  • The retriever test for filter_modules now also asserts that the filtered module is never loaded.
  • mix test: 465 passed. mix format --check-formatted: clean.

🤖 Generated with Claude Code

The retriever ran filter_modules only after a module had been loaded and
fully processed (abstract code, specs, callbacks, markdown), so on a large
project an allow-list of a few modules cost as much as documenting
everything. Autolink then checked every module reference against the list
of filtered module nodes with Enum.any?, which made a filtered build slower
than an unfiltered one.

Now the retriever applies filter_modules to the docs chunk alone, before
Code.ensure_loaded and module_data, and returns the filtered modules as
names. Autolink keeps them in a MapSet. The kept modules are loaded with one
Code.ensure_all_loaded/1 call instead of a Code.ensure_loaded/1 round trip
per module from parallel tasks, which serializes on the code server.

Measured on a project with ~50k modules (6 schedulers): a 100-module
allow-list went from 55-62 s to 22 s in the retriever+formatter phase with
the modules preloaded, and from 468 s to 20 s without a preload; the full
unfiltered build went from 473 s to 102 s without a preload. Output is
unchanged.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@KirillKayumov

Copy link
Copy Markdown
Contributor Author

Hi @josevalim , when you have time, could you please take a look?

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Comment thread test/ex_doc/retriever_test.exs Resolved
@josevalim
josevalim merged commit 82f0869 into elixir-lang:main Nov 2, 2026
6 checks passed
@josevalim

Copy link
Copy Markdown
Member

💚 💙 💜 💛 ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants